fix: restart the MQTT session when one device stops receiving messages - #1044

Draft: the final code (bab8eb2) is running on my Home Assistant now. I'll mark this ready after one or two more days of testing. Everything below is from running this branch with real devices since 2026-11-09, after I first saw the problem in early November and reported it in #1012 on 2026-11-04.

Fixes #1012.

Problem

All devices on an account share one MQTT session. On my account, the broker sometimes stops delivering a Saros 10R's (V1, roborock.vacuum.a144) MQTT responses, while an F25 Ultra (A01, roborock.wdv.a194) on the same connection keeps working. The HealthManager counted timeouts for the whole session, and any successful response reset that count. The F25's responses hid the Saros's timeouts, so the session was never restarted. In Home Assistant the map is only available over MQTT, so it stopped updating for hours. Status, battery and commands kept working because they use the local connection.

Changes

1. fix: restart the MQTT session when one device stops receiving messages. Timeouts are now tracked for each device.

   • A device with 3 timeouts within 30 minutes restarts the session. A single response in between doesn't reset the count, because a partially broken connection still delivers some responses.
   • The device counts as recovered after 3 responses in a row, or at its first response once its timeouts are older than 30 minutes.
   • 3 timeouts in a row across several devices still restart the session at once, as before.
   • Restarts are paced. There is at most one every 30 minutes. While the same device keeps failing, its interval doubles (30 min, 1, 2, 4 h, then every 6 h), so a device that is really offline doesn't keep interrupting the others.
   • Each decision is logged at info level.

2. fix: reconnect right away after a forced MQTT session restart.

   • A forced restart used to wait out the 10 s backoff, which cut every device off for 10 s. It now reconnects immediately. Other disconnects, and a failed first attempt, still use the backoff.
   • Every new connection clears the timeouts counted on the previous one.

3. feat: add MQTT health to device diagnostics. V1 and A01 diagnostics get an mqtt_health section with:

   • failing_since, last_success and last_timeout;
   • request counts;
   • the last 10 events, newest first: timeouts_started, session_restart with its cause (this_device, other_device or several_devices), and recovered.

   B01 channels don't report to the health manager yet, so they get no section.

4. docs: the health management section of docs/DEVICES.md described local connection recovery as the HealthManager. It said A01/B01 devices don't need one. I split it into local connection recovery and MQTT health management.

Size: 13 files, +880/−45. About 600 of those lines are tests.

Testing

• pre-commit run --all-files: ruff, mypy, codespell and the other hooks pass. Unit tests: 2292 passed. MQTT broker tests (-m mqtt_broker) against EMQX from compose.yaml: 11 passed.
• Home Assistant 2026.11.0, Roborock integration as a custom component pinned to this branch. Devices: Saros 10R (V1, local + MQTT) and F25 Ultra (A01, MQTT only) on one account.

What we observed

These are observations from one account, not an explanation.

• Lifetime: the Saros's MQTT replies stop roughly 3–4 hours after a connection is opened. On idle days this was very regular: 3 h, 3 h, 4 h and 4 h after the connect, to within the 10-minute probe interval. On days with cleaning it lasted longer (5 h 45 min and 6 h 20 min). In one case the connection died about 3 h after the vacuum returned to the dock, during the next cleaning.
• Scope: only the Saros's replies are lost. The client keepalive keeps passing, and the F25 keeps sending and receiving on the same connection. The Roborock app controls the Saros normally during the outage.
• Partial failures: sometimes the failure is partial. During one cleaning, every other request got a reply for about 30 minutes. With the previous "3 timeouts in a row" rule, the restart came 28 minutes late.
• Topics: they are identical before and after a reconnect. I checked the debug logs across three connections. Only the client id and the broker session change.
• What fixes it: a new connection, every time. After each forced restart the Saros answered on the next request.
• Unrelated daily blip: at about 03:38 every night, the Saros briefly drops its local connection and the F25 reports {"online": false}. It recovers by itself in a few minutes.

We can't tell whether the broker stops forwarding our requests to the device or the device's replies to us. Requests are QoS 0, so nothing confirms delivery.

Results with this branch

• Forced restart: each one reconnects in 0.16–0.20 s. The F25 had no timeouts at any of them. Over a day with 5 Saros failures, its only 2 timeouts came from a short internet outage.
• Failure during cleaning: the session was restarted 82 s after the first map timeout, and the Saros answered 31 s later.
• Saros powered off for about 2 h 20 min: that produced about 45 timeouts per 30 minutes from status requests falling back to MQTT. The session was restarted only 3 times, at +1 min, +30 min and +1 h 30 min, with growing intervals. The F25 had 150 successful requests and 0 timeouts. After power-on the Saros was marked recovered.
• Internet outage: a broker disconnect followed by DNS failures was handled by the normal backoff. The immediate attempt is used only once after a forced restart.

What didn't work, or was dropped

• Resubscribing the device's topic on the same connection after it failed (UNSUBSCRIBE + SUBSCRIBE). The Saros stayed silent for 3 more requests. Only a new connection helped.
• Resubscribing every topic every 2.5 h, before the failure, to see whether the subscription's age matters. The connection still died 3 h 10 min and 3 h 30 min after connecting, 30–60 min after a fresh resubscribe. Only the connection's age seems to matter.
• Counting timeouts in a row: partial failures kept resetting the count (see above). Replaced with the 30-minute window.
• Recovering only once the timeouts expired: after the powered-off device came back, it stayed "failing" for 30 minutes despite answering. If nothing used MQTT afterwards, that would have lasted until the next cleaning. Replaced with 3 responses in a row.
• A periodic MQTT probe (GET_NETWORK_INFO every 10 minutes for locally connected V1 devices). It found broken topics while the vacuum was idle, but that meant a restart every 3–4 hours that nobody needed. Without it, a broken topic is found by the next map requests and repaired within about 2 minutes.

Considered and not done

• One MQTT session per device. It goes against the shared-session design, adds broker connections and rate-limit risk, and the shared restart now costs other devices only ~0.2 s.
• Fetching the V1 map over the local connection. The docs say MQTT is the only way to get the map.
• A proactive reconnect every ~2.5 h. The lifetime isn't fixed (3–6 h), and the reactive restart already handles it.

Open questions

• The root cause is unknown. It is probably on the broker side: per-connection routing for one device going stale. I can't verify that from the client.
• B01 (Q7/Q10) channels don't report to the health manager. Q7 uses request/response and could be wired in the same way as A01. Q10 has no replies to time out. I didn't do either here because I have no B01 device to test with.